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Abstract— Scope of this paper is to investigate the 
performances of different identity declaration fusion algorithms in 
terms of probability of correct classification, supposing that the 
information for combination of the inferences from the different 
classifier is affected by measurement errors. In particular, these 
information have been assumed to be provided in the form of 
confusion matrices. Six identity fusion algorithms from literature 
with different complexity have been included in the comparison: 
heuristic methods such as voting and Borda Count, Bayes’ and 
Dempster-Shafer’s methods and the Proportional Redistribution 
Rule n° 1 in the Dempster-Shafer’s framework. 


Keywords—target classification, 
matrix. 


identity fusion, confusion 


I. INTRODUCTION 


In a multi-sensor system the target classification performance 
can be improved by suitably combining the inferences 
generated by the autonomous classifiers of the single sensors 
(identity declaration fusion [1]). For this purpose it is desirable 
to use the available information about the classification 
performances of the single sensors. The confusion matrix, 
whose elements correspond to the likelihood of the different 
involved classes, is a compact and detailed way of 
representing the classification performance, from which the 
Probability of correct classification (Pcc) and the probability 
relative to the various misclassification errors can be derived. 
In particular the elements of the confusion matrix can be used 
to maximize the a-posterior Pcc according to Bayes’ theory. In 
this case, if the numerical values of the confusion matrix were 
errorless, the performance of the identity fusion would be 
optimal. However, in practice, these values are estimated and 
affected by errors. In these conditions, the Bayes’ rule does 
not always produce best results. In particular, in presence of 
strong-conflicting inferences and estimation errors, the 
application of Bayes’ rule can be not effective. It can be better 
to apply simpler combination rules as some heuristic methods 
that are more robust to errors. 

Dempster-Shafer’s theory has been presented as a 
generalization of Bayes’ theory in [2]. A recent work has 
disputed this claim, limiting its correctness to the case of 
uniform a-prior probabilities [3]. The problem of the 
Dempster-Shafer rule (and of Bayes’ rule) in presence of 
conflicting inferences has been pointed out by the well-known 
Zadeh’s paradox [4]. In this paper the performances of 


different algorithms that use estimated confusion matrices 
affected by errors to combine the inferences from the single 
classifiers are investigated. In particular the heuristic methods 
based on voting and on ranking (Borda count) are compared 
with the methods using Bayes’ and Dempster-Shafer’s rules. 
Moreover the effectiveness of the redistribution of the 
conflicting masses preserving the Dempster-Shafer 
framework, like Proportional Redistribution Rules (PCR) is 
evaluated. 


The paper is organized as follows: 

e insection II the identity fusion algorithms considered 
in this paper are briefly described; 

e in section III, four simple but representative identity 
fusion problems are introduced as study cases and the 
corresponding results using different mean values of 
the estimation errors of the confusion matrices are 
reported and commented; 

e section IV gives the conclusions. 


Il. IDENTITY FUSION ALGORITHMS 


The algorithms for the identity fusion considered in this paper 
are: 


e Majority Voting (MV), 
e Weighted Voting (WV), 
e Borda count, 

e Bayes’ rule, 


e Dempster-Shafer’s (D-S) rule with the following basic 
belief mass assignment: “q-least commitment”, 


e Proportional Redistribution Rule n°l (PCRI) with the 
following basic belief mass assignment: “q-least 
commitment”. 


A brief description of the fusion algorithms follows. 


Majority voting [5] is the simplest method for the 
combination of inferences: each inferred class corresponds to a 
single vote and the selected class after fusion is the most voted 
class: all the inferences matter the same. In the modified 
weighted version, the different votes are weighted by the 
estimated Pcc of the voter/classifier. 


Voting methods use only the top choice of each classifier, 
but secondary choices often contain near misses that should not 
be overlooked. The Borda count [5] is a method in which 


classes are ranked in order of preference; it gives each class a 
certain number of points corresponding to the position in which 
it is ranked by each classifier. The class with the highest 
scoring is then selected after fusion. 


Bayes’ theorem [1,4] links the degree of belief in a 
proposition before and after accounting for evidence (a-priori 
and a-posteriori probabilities). The a-posteriori probability of a 
combination of two or more evidences is obtained by the 
multiplication of the likelihoods of the single evidences (the 
independence of the evidences is assumed). 


Dempster-Shafer theory [1,4 and 5] allows to specify a 
degree of ignorance instead of being forced to supply 
probabilities that add to unity. In this formalism a degree of 
belief (also referred to as a Basic Belief Mass - BBM) is used 
rather than a Bayesian probability distribution. BBM values are 
assigned to sets of possibilities (union of one or more classes) 
rather than to a single class, probability is instead represented 
by intervals that are lower-bounded by the value “belief” (or 
“support”) and upper-bounded by the value “plausibility”. 
BBM values from different sources can be combined with 
Dempster-Shafer's rule of combination, assuming independent 
belief sources. There are more than one possible assignment for 
transforming probabilities into BBMs [7,8 and 9]. The “q-least 
commitment” basic belief mass assignment (that corresponds 
to the maximum compatible degree of ignorance) has been 
considered in this paper to transform the CM (Confusion 
Matrix) likelihoods and the a-priori probabilities into BBM 
values. 


Proportional Redistribution Rules (PCR) is a family of 
fusion rules for the combination of uncertain information 
allowing to deal with highly conflicting sources. The PCR rules 
can be used as alternatives to the Dempster-Shafer's 
combination rule. Six PCR rules (PCRI-PCR6) have been 
defined [10,11 and 12]: from PCR1 up to PCR6 one increases 
in one hand the complexity of the rules, but in other hand one 
improves the accuracy of the redistribution of conflicting 
masses. The basic common principle of PCR rules is to 
redistribute the conflicting mass proportionally with some 
functions depending on the sum of the masses assigned by the 
single inferences. PCR1 is the least accurate combination rule 
of the PCR family, but it is the simplest to implement and it has 
been considered in this paper. PCR2-6 implementations are 
significantly more complex because the conflicting mass is 
redistributed only to the non-empty set that are involved in the 
conflict (extra computer memory is needed to keep track of the 
conflicting hypotheses and extra computation load is needed 
for combining them). A particular interesting action point for 
further investigation would be testing the most efficient PCR 
tule (PCR6) [12]. 


A. Combination rules 


In this section, the rules of Bayes, Dempster-Shafer and 
PCR1 for the combination of two classifiers are briefly 
recalled. For further details and the generalization of the rules 
with more than two classifiers, see references [1], [4], [10] and 


[11]. Voting and Borda count combinations are not considered 
here because they consists simply in the sums of respectively 
the votes and the ranks. 


Let consider a set Q of possible exhaustive and mutually 
exclusive classes C, , with N being the cardinality of this set: 
Q={C,,Cy,-Cy} (1) 
Let suppose that the independent classifiers 1 and 2 infer 
respectively the classes C; and C;; the a-posterior probability 
P,(c r/AO B) of inferring the class C, resulting from Bayes’ 


rule of combination is: 


Pale,fe, nc, )= LE IGIC AG) @) 


SAG Io) BC /C,) AIG) 


where: 


e 6©#P, (-) is the a-prior probabilities (without any 
information obtained by previous classifications) of 
the considered class; 


¢ BI(C,/C,), P(C,/C,) are the probabilities that 
classifiers 1 and 2 infer the class C,, assuming that 
the true class is C, (likelihoods). 
Let consider the power set 2° of Q as the set whose 
elements are all the possible subsets of Q: 
2° = 1h, Fy Cb ={8,C,,Cy,-+-, Cy, NOg, Cy NEw, QI 
(3) 
where @ is the empty set. The cardinality of 2° is 2”. 


Let suppose that the independent classifiers 1 and 2 assign 
respectively BBMs m,(-) and m,(-) to the elements included 
in the power set 2°; the combination BBM m,,(F,) of F, 
resulting from Dempster-Shafer’s rule of combination is: 

m,(F;)- my(F; ) (4) 


my (Fy) = in 


i,j / Fy=F, OF; 
where m, is the global conflicting mass, defined as follow 


m, = Ym, (F,,)-ms(F, (5) 


pq! Fy OF, =0 
In the case of the PCR1 rule the combination BBM 
my(F;,) of F,, is instead: 
m(F,)-m,(F,)4 m,(F;,) + my (F;,) ae 
¢ hk 


2N (7 


(m,(F,)+ mp (F;,)) 


my (Fx) = 
i,j FOF; =F, #0 


h=1 


(6) 


Ill. STUDY CASES 


Four simple but representative study cases (three different 
classifiers for a three classes problem) have been investigated, 
as follows: 


¢ complementary confusion matrices, 

e¢ supplementary confusion matrices, 

¢ complementary conflicting confusion matrices, 
e supplementary conflicting confusion matrices. 


By complementary CMs it has been meant that the single 
classifiers show a complementary expertise in the recognition 
of the different classes. By supplementary CMs the different 
classifiers show similar behaviors. By conflicting CMs a 
possible overestimation of the performance of the single 
classifiers can make harder an effective combination of the 
contradictory inferences from different classifiers when they 
occur. A quantitative definition of complementary and 
supplementary CM can be found in [13]. 


In the following sub-sections, the estimated confusion 
matrices that have been selected for the four study cases are 
reported. The columns of the matrices represent the true 
classes, while the rows correspond to the inferred classes, so 
the element (k,h) of a matrix is an estimation of the probability 
of declaring k" class when the true class is the h" one: 


M (k,h)= P.(D=k/T =h) (7) 


The a-prior probabilities of the different classes are 
assumed equal. A block diagram of the fusion system is shown 
in fig. 1. 


The performances of the six algorithms in correspondence 
of the identity fusion of six inferences (two independent 
inferences for each classifier) have been considered. The 
performances of the different algorithms have been computed 
with 1000 Monte Carlo trials, each generating independent 
samples of the true confusion matrices and a-prior 
probabilities. 


The results of the Monte Carlo trials are represented by the 
curves corresponding to Empirical Cumulative Distribution 
Function (ECDF) versus the Pcc. The x-axis values of Pcc have 
been computed exactly, that is the contribution of all the 
possible permutations of the single sensor inferences has been 
considered. 


A. Complementary confusion matrices 


The following confusion matrices corresponding to three 
different classifiers have been considered: 


0.90 0.30 0.30 
M,=|0.10 0.40 0.30 


0 0.30 0.40 

10.40 0.30 0.10] 
M,=|030 0.40 0 (8) 

10.30 0.30 0.90 | 

[0.40 0 0.30] 
M,;=|0.30 0.90 0.30 

10.30 0.10 0.40 | 


The performance is dependent on the true target class (class 
1, 2 or 3): 


e the first classifier identifies correctly targets 
belonging to class 1 (on average it makes only one 
mistake in ten of its inferences), while it almost 
randomly infers in correspondence of targets 
belonging to class 2 or class 3, 


e the second classifier identifies correctly targets 
belonging to class 3 (on average it makes only one 
mistake in ten of its inferences), while it almost 
randomly infers in correspondence of targets 
belonging to class | or class 2, 


e the third classifier identifies correctly targets 
belonging to class 2 (on average it makes only one 
mistake in ten of its inferences), while it almost 
randomly infers in correspondence of targets 
belonging to class | or class 2. 


The performances of the six different algorithms are 
reported in the fig. 2 and 3 in correspondence of an estimation 
of the confusion matrix by using respectively 30 and 10 
independent samples for each class. The performances of the 
single classifiers correspond to the dotted curves (indicated as 
Cl, C2 and C3 in the legends of the figures). Bayes’ rule, 
Dempster-Shafer’s rule, Borda count and PCRI1 give similar 
results, the performance of PCRI is barely the best. The voting 
algorithms present significantly worse performance. 


B. Supplementary confusion matrices 


The following confusion matrices corresponding to three 
different classifiers have been considered: 


0.70 
M, =| 0.20 
0.10 
| 0.80 
M, =| 0.10 
| 0.10 
| 0.60 
M, =| 0.20 
| 0.20 


0.10 
0.70 
0.20 
0.10 
0.80 
0.10 
0.20 
0.60 
0.20 


0.20 
0.10 
0.70 
0.10 | 
0.10 (9) 
0.80 | 
0.20 | 
0.20 

0.60 | 


In the second example, three classifiers with supplementary 
confusion matrices have been selected. A single classifier can 
recognize all the three classes with the same accuracy, but the 
accuracy differs from classifier to classifier: 


e the first classifier has an estimated probability of 
correct classification equal to 70%, 


e the second classifier has an estimated probability 
of correct classification equal to 80%, 


e the third classifier has an estimated probability of 
correct classification equal to 60%. 


The performances of the six different algorithms are 
reported in the fig. 4 and 5 in correspondence of an estimation 
of the confusion matrix by using respectively 30 and 10 
independent samples for each class. The performances of the 
single classifiers correspond to the dotted curves (indicated as 
Cl, C2 and C3 in the legends of the figures). All the algorithms 
give comparable performance. PCRI and Bayes’ rule 
performance are exactly the same and they are slightly better 
than the others, weighted voting performs better than Borda 
count and majority voting. 


C. Complementary conflicting confusion matrices 


The following confusion matrices corresponding to three 
different classifiers have been considered: 


1.00 0.00 
M, =|0.00 0.60 
0.00 0.40 
[0.60 0.00 
M, =|0.00 1.00 
10.40 0.00 
[0.60 0.40 
M,=/|0.40 0.60 
10.00 0.00 


0.00 
0.40 
0.60 
0.40 | 
0.00 (10) 
0.60 | 
0.00 | 
0.00 
1.00 | 


In this third example, the three classifiers can be affected by 
conflicting inferences. As consequence, the application of 
Bayes’ rule to fusion leads to severe performance degradation 
with respect to heuristic methods. The problem arises from an 
overestimation of the performance of the single classifiers. 


The performances of the six different algorithms are 
reported in the fig. 6 and 7 in correspondence of an estimation 
of the confusion matrix by using respectively 30 and 10 
independent samples for each class. The performances of the 
single classifiers correspond to the dotted curves (indicated as 
Cl, C2 and C3 in the legends of the figures). PCR1 gives the 
best result that is slight better than Borda count. Majority and 
weighted voting have coincident performance that are 
significantly worse than the one of PCR1. The performance of 
Bayes’ rule and Dempster-Shafer rule are perfectly coincident 
and worse than all the others because of the presence of 
conflicting inferences. 


D. Supplementary conflicting confusion matrices 


The following confusion matrices corresponding to three 
different classifiers have been considered: 


1.00 0.00 0.00 
M,=M,=M,=|0.00 1.00 0.00 (11) 
0.00 0.00 1.00 


In the forth example, three classifiers with identity 
confusion matrices as estimations have been selected: 
according to these estimations the single classifier is never 
wrong. If the classifiers disagree on the inferred class, the 
Bayes’ rule of fusion leads to severe performance degradation 
with respect to heuristic methods. 


The performances of the six different algorithms are 
reported in the fig. 8 and 9 in correspondence of an estimation 
of the confusion matrix by using respectively 30 and 10 
independent samples for each class. The performance of the 
single classifiers correspond to the dotted curves (indicated as 
Cl, C2 and C3 in the legends of the figures). The performances 
of the voting algorithms, Borda count and PCR1 are perfectly 
coincident and near to 100%. The performances of Bayes’ rule 
and Dempster-Shafer rule are perfectly coincident and much 
worse than all the others because of the presence of conflicting 
inferences, even much worse than the performance of the 
single classifiers. 


E. Summary results 


In table I the average (over the 1000 Monte Carlo trials) 
Pcc is reported for all the investigated study cases. It has been 
reported also an intermediate case where 60 samples (20 
samples per class) for the estimation of each confusion matrix 
have been considered. It can be noted than PCR1 always brings 
the highest Pcc of all the six considered combination rules. 


IV. CONCLUSIONS 


Simulation results show that in the considered study cases, 
the algorithm using the PCRI1 rule of combination brings the 
best performance of all the six considered alternatives and 
definitely overcomes Bayes’ and D-S’s rules in the cases where 
the probability of conflicts between the inferences is high. This 
performance difference increases with the decrease of the 
number of samples used for the estimation of the confusion 
matrices. This behavior is a consequence of the poor 
performance of the latter two combination methods in presence 
of conflicting inferences from the different classifier, as 
claimed by the Zadeh’s paradox. In two investigated study 
cases with conflicting confusion matrices Bayes’ and 


Dempster-Shafer rules perform even worse than heuristic 
approaches. 


In the cases where the conflict is less likely probable the 
performance of the PCRI is comparable with the ones of 
Bayes’ and D-S’s rules (the same or slightly better). 


The implementation of PCRI1 slightly increases the 
computational complexity of D-S’s rule. Future work may be 
addressed to the comparison of the performance resulting by 
the application of more complex PCR rules to the inferences of 
classifiers whose accuracies are represented by conflicting 
confusion matrices. 
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(estimation from 10 samples per class). 
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Fig. 9. ECDF of Pcc with supplementary conflicting confusion matrixes 
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ECDF of Pcc with supplementary conflicting confusion matrixes 


(estimation from 30 samples per class). 


TABLE I. SUMMARY RESULTS (N IS THE TOTAL NUMBER OF SAMPLES). 

PROBABILITY OF CORRECT CLASSIFICATION (mean value, %) 
Study MAJORITY WEIGHTED BORDA BAYES DEMPSTER PCRI1 
cases VOTING VOTING COUNT SHAFER 

N=30 | N=60 [| N=90 | N=30 | N=60 | N=90 | N=30 | N=60 | N=90 [| N=30 | N=60 | N=90 | N=30 | N=60 | N=90 | N=30 | N=60 | N=90 

Compl. 67.8 | 72.4 | 73.9 | 67.8 | 72.4 | 73.9 | 73.9 | 78.9 | 80.7 | 73.1 | 79.3 | 81.6 | 73.0 | 79.4 | 81.6 | 74.8 | 80.5 | 82.4 
CMs 
Supp. 82.3 83.9 : 9 | 82.7 | 87.4 | 88.8 : . i : : ; 90.4 
CMs 
Compl. 87.7 : ; : : . 98.1 ; : : : F : : j 99.2 
confl. 
CMs 
Supp. 98.8 : ; ; 99.8 : : : : : : . . 99.9 
confl. 
CMs 


